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SOME  PECULIAR 
SEMI-MARKOV  PROCESSES 

WALTER  L.  SMITH 
University  of  North  Carolina 

Summary.  Experience  with  semi-Markov  processes  with  finite  expected  waits 
suggests  that  the  behavior  of  Markov  processes  is  a  good  guide  to  understanding 
the  behavior  of  the  more  general  process.  However,  examples  are  given  to  show 
that  when  expected  waits  are  infinite  quite  surprising  behavior  is  possible.  For  a 
two-state  aperiodic  semi-Markov  process  the  instantaneous  state  probabilities 
Pi(t )  can  have  (C,  l)-limits  but  not  strict  limits;  for  a  three-state  (and  irreducible ) 
process  one  can  have  Po(t)  tend  to  a  strict  limit  as  t  — » °o  but  Pi(t)  and  P2(t ) 
not  even  have  ( C ,  l)-limits.  For  an  aperiodic  irreducible  infinite  chain  one  can 
have  Pi(t)  — >  Ti  >  0  as  t  — >  a>,  for  every  i,  yet  2  71- ,•  <  1. 

1.  Introduction 

Semi-Markov  processes  were  introduced  simultaneously  by  L6vy  [3]  and  by 
Smith  [7],  [8].  The  constructive  definition  of  Smith,  which  is  valuable  so  long 
as  only  a  few  states  are  instantaneous,  has  been  given  an  elaborate  and  formal 
treatment  by  Pyke  [5],  [6]. 

For  the  present  note  we  shall  suppose  that  we  are  given 

(i)  the  transition  matrix  ||p;,||  of  an  irreducible  and  recurrent  Markov  chain 
of,  possibly,  infinitely  many  states  ; 

(ii)  a  sequence  of  proper  distribution  functions  of  nonnegative  random 

variables  and  such  that  there  is  at  least  one  i  such  that  0,(0 -f)  <  1. 

We  imagine  the  process  develops  as  follows.  An  initial  state,  say  i0,  is  selected, 
and  the  process  stays  in  this  state  for  a  period  of  time  governed  by  the  distribu¬ 
tion  function  i2*0(rc).  At  the  end  of  this  wait  in  the  state  io  the  process  then  selects 
a  fresh  state  in  accordance  with  the  transition  matrix  ||pfy|| ;  thus,  with  proba¬ 
bility  Pi^  the  system  now  moves  to  state  i\.  Having  reached  state  i\  the  system 
waits  there  a  period  of  time  governed  by  12*, (2;),  and  so  on.  It  is  assumed  that 
successive  waits  are  independent.  Under  the  assumptions  we  are  presently 
making  (especially  the  recurrence  of  ||p;/||)  there  will  be,  with  probability  one, 
finitely  many  transitions  in  any  finite  time  period.  To  avoid  ambiguity  we  may 
suppose  that  at  the  instant  of  a  transition  the  system  is  in  the  state  in  which  it 
will  next  reside  for  a  strictly  positive  amount  of  time. 

For  purposes  of  discussion,  let  us  suppose  (with  no  loss  of  generality)  that  the 
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initial  state  is  0,  and  let  us  write  Pft )  for  the  probability  that  the  system  is  in 
state  i  at  time  t.  We  shall  also  suppose  that  the  semi-Markov  process  is  aperiodic; 
that  is,  the  time  intervals  between  successive  returns  to  state  0  are  not,  with 
probability  one,  multiples  of  some  fixed  span  >  0.  Subject  to  these  under¬ 
standings,  and  when  the  distribution  functions  {12,(2:)}  all  have  finite  mean 
values,  Smith  [7]  showed  that  the  probabilities  P%{t)  all  tend  to  limits: 
Piit)  — » 7 r,-  >  0  as  t  — >  oo .  In  this  respect,  and  in  many  others,  semi-Markov  proc¬ 
esses  behave  so  very  much  like  Markov  processes  that  one  might  be  excused  for 
supposing  them  unworthy  of  study.  For  instance,  much  of  the  interest  in  Markov 
processes  arises  from  the  existence  of  instantaneous  states,  and  it  seems  unlikely 
that,  were  one  to  adopt  a  suitably  changed  definition  for  the  semi-Markov 
process  so  as  to  allow  an  abundance  of  instantaneous  states,  the  resulting  more 
“general”  processes  would  exhibit  any  behavior  not  an  obvious  copy  of  that 
already  found  in  familiar  Markov  processes.  However,  there  is  one  possibility 
for  semi-Markov  processes  which  is  precluded  from  Markov  processes;  this  is 
the  possibility  that  the  waits  in  the  states  may  have  infinite  expectation. 

In  a  very  short,  but  interesting,  paper  Derman  [1]  considered  a  semi-Markov 
process  of  only  two  states,  0  and  1  say,  in  which  p0,o  =  pi,i  =  0.  He  showed 
that,  if  we  denote  Laplace-Stieltjes  transforms,  for  real  s  >  0, 


(1.1) 

then 

(1.2) 


12*  (s)  =  Jq_  e~sx  dili{x), 


lim 

T— »« 


If 

TJ  o 


Pi(t)  dt  —  lim  — 
«-»o+  i 


1  -  12  t(s) 

-  12!0)12I(s) 


if  either  of  the  limits  exist.  In  other  words,  (C,  l)-limits,  at  least,  can  exist  for 
the  probabilities  Pft)  even  when  wait-distributions  have  infinite  means. 

Derman’s  result  raises  a  number  of  questions  about  semi-Markov  processes. 
Do  strict  limits,  as  opposed  to  (C,  l)-limits,  exist  for  the  probabilities  P*(0? 
Do  semi-Markov  processes  of  infinitely  many  states  exhibit  similarly  pleasant 
asymptotic  behavior?  In  another  paper  we  shall  discuss  sufficient  conditions 
for  the  existence  of  strict  limits  to  the  Pft)  when  the  &i(x)  have  infinite  means. 
In  this  paper  we  shall  exhibit  examples  of  semi-Markov  processes  for  which  the 
asymptotic  behavior  of  the  Pft)  is  disturbing,  if  not  surprising;  the  examples 
serve  to  show  that,  unlike  what  happens  with  semi-Markov  processes  with 
finite  mean  waits,  when  the  mean  waits  are  infinite  analogy  with  ordinary 
Markov  processes  is  quite  useless. 


2.  A  two-state  process  with  no  (C,  l)-limits  to  the  Pft)  as  t  — >  <x> 

Consider  the  Derman  result  (1.2).  We  have 


1  -  12*(s) 

1  -  0t(s)12|(s) 


1 


1  +121 


-  asp?) v 

- 12100/ 


(2.1) 
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and  since  — >  1 

existence  of 

(2.2) 
is  that 
(2.3) 


as  s  — >0+,  a  necessary  and  sufficient  condition  for  the 
lim  ^  f  Pi(t)  dt 

T — >»  1  Jo 

V  1  -  fil(s) 

*— >0+  1  -  ^l(s) 


should  exist.  At  this  point  the  following  theorem  is  relevant. 

Theorem  1.  If  F(x)  and  G(x)  are  distribution  functions  of  nonnegative  random 
variables  and  X  >  0  is  some  finite  constant,  then  a  necessary  and  sufficient  condition 
to  ensure  that 

(A)  Jj  (1  -  F(y)}  dy~\  f*  { 1  -  G(y)}  dy  as  x  -»  oo , 

is  that 


(B) 


{1  —  F*(s)}  ~  X  (1  —  G*(s)}  as  s  — >  0+. 


Proof.  This  theorem  is  in  fact  an  easy  consequence  of  some  general  Abelian- 
Tauberian  results  of  Feller  [2].  However,  it  is  easy  if  F(x)  has  a  finite  mean, 
and  it  is  also  easy  to  show  that  if  F(x)  has  an  infinite  mean  (so  that  the  left-hand 
member  of  (A)  diverges  as  £—>«),  then  (A)  implies  (B).  The  only  really 
difficult  part  is  to  prove  that  (B)  implies  (A)  when  F  has  an  infinite  mean  and 
it  is  of  interest  to  see  that  a  slight  modification  of  the  classical  Karamata  argu¬ 
ment  (see,  for  example,  Widder  [9])  can  accomplish  what  we  need. 

Define  h{ x)  =  0  for  x  <  h(x)  =  1  for  x  >  1.  Choose  a  small  e  >  0  and  let 
P(x)  be  any  polynomial  such  that  P(0)  =  0,  P(l)  =  1,  and 


(2.4) 


fw-wa,  n, 

Jo  X  \log  2  |log  x\) 


<  e. 


It  is  not  difficult  to  see  that  this  can  be  done.  Then,  if  we  assume  (B),  and  note 
particularly  that  P(0)  =  0,  as  s  — » 0+ 


(2.5) 

From  this, 
(2-6) 


as  T 


X  fj  P(e-){  1  -  <?«)}  dt  ■  '  JJ  P(e-‘){  1  -  F(t)}  dt. 
by  obvious  maneuvers,  we  have 


G(t)}  dt  +  XT  {pfy-—htyl  |l  -  G  (j  log  dx 

~  /:  m ^  r {nx)  i  a(j:)}  {>  -  ^  (ri°^)} 

But,  for  0  <  *  <  1, 

r{i-G(riogi)}  j 

r!(i-e(0}  dt-W 


dx 


(2.7) 
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while,  for  \  <  x  <  1,  a  similar  inequality  holds  if  we  replace  the  right  member 
by  l/(log  l/x).  Thus,  in  view  of  (2.4), 


(2.8) 


{P(x)  -  h(x)} 


(1  -G(Tlog  l/x)} 


<  € 


/. 


nog  2 


{1  -  G(t)}  dt 


Clearly  a  similar  inequality  holds  with  F  in  place  of  G,  and  the  theorem  follows 
from  (2.6). 

With  the  aid  of  theorem  1  and  Derman’s  result  we  can  easily  construct  an 
example  of  the  sort  required  in  this  section.  We  have  only  to  construct  distribu¬ 
tion  functions  tti(x)  and  ft2(z)  such  that 


(2.9) 


—  &i(x)}  dx 
JqT  {1  —  02(*)}  dx 


does  not  tend  to  a  limit  as  T  —>  <» .  Such  a  task  is  too  easy  for  words. 


3.  A  two-state  aperiodic  process  with  (C,  l)-limits  but  not  strict 
limits  to  the  P,(0 

We  consider  the  basic  two-state  process  of  Derman  once  more  (p0.o  =  Pi,i  =  0) 
and  shall  arrange  that 


tends  to  a  limit  as  T  —>  <x> ,  while  P\{t )  does  nothing  of  the  sort.  It  is  vital,  of 
course,  to  note  that  we  are  not  relying  on  any  periodicity  to  achieve  our  ends. 

Suppose  that  0i(0+)  =  0  and  that,  for  some  large  £  >  0,  both  O0 (x)  and 
Qi(x)  have  been  defined  for  all  x  <  £.  Suppose  further  that 

(3.2)  fi0(£)  <  1,  fii(£)  <  1. 

Let  us  write 

(3.3)  OK*)  =  1  -  fl<(ar),  i  =  0,  1, 

and 

(3.4)  ,,  =  min  {fi^(£  -  0),  S2f(£  -  0)}. 

Choose  t0  £,  but  unspecified  for  the  moment.  Define 

(3.5)  Oo(z)  =  Oifc)  =  v  for  all  £  <  x  <  to, 

and  set 

(3.6)  F{x)  =  Jj  Q0(x  —  z)  dUi(z). 

Let  H(x )  be  the  renewal  function  based  on  the  distribution  function  F(x).  If 
we  assume  there  was  a  transition  into  state  0  at  t  =  0,  then 
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(3.7) 


Po(0  —  0o(0  “l-  JQ  0$(t  —  r)  dH{r). 


Suppose  we  introduce  improper  distribution  functions  as  follows : 


(3.8) 

tto(x)  = 

fSJoW, 

U, 

V  Al 

(3.9) 

= 

mx), 

u, 

x  <  £, 
x  >  £, 

(3.10) 

F{x)  = 

—  z)  d£L\(z). 

Then  F(oo )  =  (1  —  t?)2  and  if  U(x)  is  the  renewal  function  based  on  the  improper 
distribution  function  F(x),  then  Hix)  f  ((1  —  t))2/  1  —  (1  —  17) 2)  as  x  f  00. 

Choose  T  >  £  such  that  77(»)  —  H(T)  <  77  and  take  to  >  2T.  Then 
H(x)  =  7J(x)  for  all  x  <  t0.  Furthermore,  we  can  choose  t0  sufficiently  large 
such  that 


(3.11) 

differs  from  unity  by 
Suppose  we  set 


JQ  {1  —  ^0(2)}  dz  +  tor]  +  Trj 

j0  (1  —  ^1(2)}  dz  +  tor]  +  2Tr] 
less  than  17. 


(3.12) 


R(x)  = 


f*  (1  -  ^0(2)}  dz 
J*  {1  -  81(2)}  dz 


and  suppose  |1  —  R(£)\  <5,  say.  Then,  if  we  define 


(3.13) 


it  will  follow  that 
(3.14) 


&o(x)  —  2V7 

ni(x)  =  v, 


|1  —  i?(x)|  <  max  (5, 77) 


t0<x<t0  +  2T 
in  the  same  range, 


for  all  x  in  the  range  £  <  x  <  t0  +  27\ 

We  have  introduced  a  probability  \r]  at  x  =  to  in  the  construction  of  £20. 
Thus,  it  is  not  true  that  H(x )  =  U(x)  for  to  <  x  <  to  +  2T.  However,  for 
0  <  x  <  2T  we  do  have  (recall  t0  >  2T) 

H(x  +  to)  =  TJ(x  +  to)  +  £77  Jj  71  (x  -  z )  dQi(z) 


<  TI(x  +  to)  4-  hH(x). 


(3.15) 
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(3.16)  Po(U  +  271)  <  + 


to+2T 


&o(to  +  2T  —  z)  dH(z ) 


But, 

(3.17) 


to+2T 

to+T 


f2T  fto  fto+T  f 

<h+  +  +  + 

Jo  J2T  Jto  Jto 

^  i  i  i  r  (i  —  »7)2  I  .  , 

<  «  +  2  Lr -”(i-  +  ’ 

+  r,{H(t o  +  T)  -  H(h)}  +  {H(h  +  2 T)  -  H(t>  +  T)}. 
H(t0  +  T)  —  H(h)  <  H(T  +  (o)  +  ivE(.T)  -  11(h) 

-”  +  h  [l  -(!-,)•} 


Also, 

(3.18)  H(U  +  2T)  -  H(t0  +  T ) 

=  H(fo  +  2T)  +  f‘‘+2TTl(t<,  +  2 T-z)  da,(z) 

-  E(h  +T)-b>  JJ+T  E(to  +  T  -  z)  dili(z) 

<v  +  iv  f*  (B(k  +  2  T-z)-  H(h  +  T  -z)}  dShb) 
<v  +  W( l  -  v)- 

Thus, 

(3.19)  P0(to  +  2 T)  <  A(v), 
say,  where 

(3.20)  A  6?)  =  f  ?/  +  2r?2  -f  \ri2{  1  —  r?)  +  +  t72)  ^  q  —  7?)j' 


We  note  that  A (77)  — »  \  as  ??  j  0.  Thus  we  can  continue  our  constructive  process 
in  a  sequential  manner  and  ensure  that 


(3.21) 
However, 

(3.22) 


as  x  — *  00 ,  so  that, 
(3.23) 


lim  inf  P0(t )  < 

t—>az 


[*  (1  “  ^0(2)}  dz 

^ - >  1 

JJ  (1  -  Gi(z)}  dz 

i  f  PM  dl^i 


as 


00 . 
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By  modifications  to  the  above  argument  (we  omit  details),  one  can  also 
demonstrate  the  existence  of  a  two-state  semi-Markov  process  such  that 


(3.24) 
but 

(3.25) 


J,  f  Pi(l)  dt  — >  1 

(eh) m  =  °- 


as  T  — >  oo , 


4.  A  three-state  semi-Markov  process  in  which  P0(t ) 
Pi(t)  and  Pz(t)  do  not  even  have  (C,  l)-limits 


>  0  while 


Choose  two  distribution  functions  &i(x)  and  f22(ar)  such  that 

JqT  ftf(rr)  dx 


(4.1) 

does  not  tend  to  a  limit  as  T  - 
process  have  transition  matrix 


(4.2) 


JJ  ft2(z)  dx 

00 .  Define  fto(z)  =  |£2i(:c)  -f-  |ft2( x )  and  let  the 


Po.o  =  0 

P0.1  =  h 

P0.2  =  h 

P1.0  =  1 

II 

0 

II 

0 

P2.0  =  1 

P2,l  =  0 

P2,2  =  0. 

From  the  point  of  view  of  state  1  the  other  two  states  can  be  pooled  together 
into  a  single  state  Aa,  say,  with 


(4.3) 


0S(«)  = 


0$(«) 


1  -  fi$(s)0?(s) 


For  Pi(t )  to  have  a  ( C ,  l)-limit  we  need  (1  —  fi*(s)/l  —  fi£(s)S2*($))  to  approach 
a  limit  as  s  J,  0.  It  is  a  matter  of  computation  to  show  that  such  a  limit  does 
not  exist,  since  (1  —  fi2(s)/l  —  fl*(s))  will  not  (by  construction)  approach  a 
limit.  Similarly  P2(0  cannot  have  a  (C,  l)-limit. 

Suppose  a  transition  occurred  into  state  0  at  t  =  0.  Then 

(4.4)  Po(0  =  at i(i)  +  J'  0 S(l  -  z)  dH(z) 

where  H  (t)  is  the  renewal  function  based  on  the  distribution  function 

(4.5)  J*  Q0(£  —  z)  d%{z). 

Thus, 

(4.6)  P*0(s)  =  [1  -  S28(s)]  + 


128(s) 


1  +  S2S(«) 
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We  can  easily  arrange  that  £l0(x)  shall  have  a  nonnull  absolutely  continuous 
component.  This  will  ensure  that 

(4.7)  inf  |1  +  J2g(8)|  >0,  Re  s  >  0, 

and  we  can  then  deduce  from  a  Mercerian  theorem  (Pitt  [4],  theorem  11, 
p.  118),  that  Pott)  is  of  bounded  variation.  But  P$(s)  — >  §  as  s  j  0-}-.  Thus, 
Po(0  — »  |  as  t  — >  oo ,  and  this  demonstration  is  complete. 

5.  A  semi-Markov  process  with  infinitely  many  states,  an  irreducible 
transition  matrix,  such  that  P,(0  — *  *,•  >  0  as  t  — » oo ,  for  all  f,  but 
Z  «i  <  i 

Let  R(x)  be  the  distribution  function  of  a  rectangular  distribution  over  (0,  1). 
Let  U(x)  =  P{0  <  x)\  let  {pn},  n  =  1,  2,  3,  •  •  ■  ,  be  probabilities  such  that 
X  Pn  =  1  and  X  npn  =  oo .  Define 

(5.1)  G(x)  =  X  pnU(x  -  n ). 

n  =  1 

Suppose  the  process  is  initially  in  state  0  and  remains  there  for  a  wait  with 
distribution  function 

(5.2)  fX  G(x  —  z )  dR(z). 

At  the  end  of  this  wait  let  the  system  move  to  state  un”  with  probability  pn 
(■ n  >  0),  and  remain  there  for  a  wait  with  distribution  function  G(x  —  n), 
followed  by  two  independent  waits  with  distribution  function  R(x).  After  these 
three  waits  the  system  returns  to  state  0  and  the  cycle  repeats  itself,  and  so  on. 
Between  sojourns  in  state  0  the  system  spends  periods  outside  state  0  with 
distribution  function  A«(x),  say,  where 


(5.3) 

A-a(x')  — 

fj  Ap(x  -  z)  dR2(z), 

(5.4) 

Ri(x)  = 

Jj  R(x  -  z)  dR(z), 

(5.5) 

Ap(x)  = 

00 

X  pnG(x  -  n) 

n  =  1 


=  j‘  G(x  -  z)  dG(z). 

From  all  this  we  can  show 

(5.6)  Po(s )  =  [1  -  R*(s)G*(s )]  +  i  +  [r*(s)G*(s)]  +( [P*(s)G*(s)]2' 

If  1  +  8  +  52  =  0,  then  |5|  =  1,  and  so  the  presence  of  R*(s)  ensures  that 
inf  1 1  +  [P*(s)(?*(s)]  +  [P*(s)(j*(s)]2|  >  0  in  the  region  Rs  >  0.  Thus,  by  the 
same  Mercerian  argument  as  before  we  can  deduce  that  Pott)  is  of  bounded 
variation  and  P0(i)  — » |  as  t  — >  » . 
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On  the  other  hand,  one  can  show  by  slightly  more  involved,  but  essentially 
similar,  calculations  that  Pn(t )  — >  \pn  as  t  — »  » .  Thus, 

(5.7)  I  W  =  H!IP»  =  h 

n  =  0  n  =  1 
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